Anxiety associated with perceived uncontrollable stress enhances expectations of environmental volatility and impairs reward learning

Unavoidable stress can lead to perceived lack of control and learned helplessness, a risk factor for depression. Avoiding punishment and gaining rewards involve updating the values of actions based on experience. Such updating is however useful only if action values are sufficiently stable, something that a lack of control may impair. We examined whether self-reported stress uncontrollability during the first wave of the COVID-19 pandemic predicted impaired reward-learning. In a preregistered study during the first-wave of the COVID-19 pandemic, we used self-reported measures of depression, anxiety, uncontrollable stress, and COVID-19 risk from 427 online participants to predict performance in a three-armed-bandit probabilistic reward learning task. As hypothesised, uncontrollable stress predicted impaired learning, and a greater proportion of probabilistic errors following negative feedback for correct choices, an effect mediated by state anxiety. A parameter from the best-fitting hidden Markov model that estimates expected beliefs that the identity of the optimal choice will shift across images, mediated effects of state anxiety on probabilistic errors and learning deficits. Our findings show that following uncontrollable stress, anxiety promotes an overly volatile representation of the reward-structure of uncertain environments, impairing reward attainment, which is a potential path to anhedonia in depression.

• The preregistration stated linear mixed-models but we used logistic mixed-model to study the effects of the factors on accuracy.
• The preregistration stated that we would include game as a fixed-effect for the GLMM analysis of the reversal task, but this factor was not included.
• We used exploratory mediation analysis to study the relationship between significant factors in the GLMM.
• We deviated from the preregistration and augmented the base Reinforcement Learning Model (RLM) with a forget parameter  (0 <  < 1).
• We did not include a model with confidence modulating the softmax inverse temperature parameter.
• We did not test the hypotheses that we preregistered for the RLM models and instead used an exploratory Hidden Markov Model to characterise the effects of the factors on observed choices.
• Instead of using RStan, we used HBI toolbox to fit and compare the models.
• Instead of testing models in which individual-level parameters drawn from the grouplevel normal distributions were allowed to vary according to the subject score on perceived controllability, we tested for the correlation with the parameters outside the model.

Exploratory factor analysis
As preregistered, we initially used Exploratory Factor Analysis (EFA) to derive latent factor scores within and across questionnaires.EFA was performed using the "fa" function in the "psych" R package 67 to find latent factors present within and across these questionnaires.
The Kaiser, Meyer, Olkin (KMO) measure of sampling adequacy was close to 1 (KMO = 0.96) suggesting that the sum of partial correlations was not large relative to the sum of correlations, meaning that the data should result in distinct and reliable factors (Kaiser, 1970).
Moreover, Bartlett's test was significant (χ2 = 18801, df = 2346, p < .001),indicating that the residual correlations were all zero, also suggesting that the data was appropriate for EFA.
Because the data were not normally distributed (Shapiro-Wilk mvw = 0.98, p < .001)we used the ordinary least squares method to find the minimum residual solution ("minres" method) which provides results similar to the maximum likelihood method, without assuming multivariate normal distribution 3 .We used oblique ("oblimin") rotation because we expected our factors to be correlated.
There are many possible ways to determine the appropriate number of factors in an EFA (see 3,4 ).As we did not preregister this detail, we tested different approaches.We first ran an EFA with eight-factors, then seven, then six, and lastly five-factors.All four solutions provided a similarly good fit to the data (e.g., 5 ) with root mean square error of approximation (RSMEA) of 0.057 (five-and six-factors); 0.058 (seven-factors) and 0.061 (eight-factors).The clearest, meaningful, yet parsimonious structure was a seven-factor model, which was consistent with the parallel analysis 6 which compares the scree of factors of the observed data with a random data matrix of the same size to estimate the appropriate number of factors 3 .Factor scores were obtained using the Thurstone regression method 7,8 .However, these EFA scores were not reliable as assessed on the 49 participants who completed the scales twice 3 day apart (ICC 9,10 ranged between 0.14 and 0.61; see Table S

.1).
This was surprising as the sum scores for each scale and subscale showed good test-retest reliability (ICC range: 0.75 to 0.95).

Congeneric factor analysis
As the use of sum scores is not recommended, we deviated from the preregistration and calculated congeneric factor scores, using confirmatory factor analysis to produce weighted scores based on previously established structure of the questionaires 11 .In a congeneric model, items' contribution to the score depends on how related the item is to the construct.
Each item is allowed unique error variance, and is constrained to have a variance equal to 1 and the intercept to 0. For all scales (see Table S.3), congeneric models were a better fit to the data than parallel models (equivalent to sum scores with equal contribution for all items), indicating that the weighted congeneric scores were preferred over sum scores to be used in subsequent analyses.We also determined whether each scale should be subset into previously established subscales.
For the Perceived Stress Scale, a two-factor model was a better fit to our data than a onefactor model, in line with previous findings [12][13][14][15][16] .Uncontrollable Stress (also referred to as Perceived Helplessness in the previous literature) includes six negatively-framed items regarding the impact of perceived uncontrollable stress (e.g., "In the last month, how often have you been angered because of things that were outside of your control?").Lack of Selfefficacy includes four positively-framed items regarding perceived ability to cope with stressors (e.g., "In the last month, how often have you been able to control irritations in your life?".The two stress factors significantly correlated with one another, r(425) = 0.636, p < .001.For the Perceived Risk of COVID-19 scale, a two-factor model (Perceived Likelihood and Perceived Severity of COVID-19 Risk) was a better fit than a one-factor model.The two COVID-19 factors significantly correlated with one another, r(425) = 0.506, p < .001.Although the PHQ-9 is often considered to be a one-factor construct, a two-factor structure with a cognitive and affective factor, and a somatic factor can also be a good fit (e.g., 17 ), and was a better fit to our data than a one-factor model.But because the two factors were so highly correlated, r(425) = 0.772, p < .001,for the sake of parsimony, we treated Depression as one factor.
The State and Trait Anxiety Inventory scales can be further divided into negatively-framed items indexing the presence of anxiety symptoms (e.g., "I feel upset"), and positively-framed items indexing the absence of anxiety symptoms (e.g., "I feel calm") which were reversecoded so higher scores indicate, for example, less calm.Consistent with previous work 18,19  Econometrica 57, 307 (1989).

Mediation analysis
The fact that both Trial x Uncontrollable Stress and Trial x State Anxiety interactions were significant in separate models, but only the State Anxiety interaction was close to significant in the combined model suggests that variability in State Anxiety may explain the effect of Uncontrollable Stress on learning.To test this possibility, we implemented an exploratory (not preregistered) causal mediation analysis (Imai et al., 2010) with Uncontrollable Stress as the predictor and State Anxiety as the mediator (Table 2).To obtain a summary measure of learning we extracted random slopes (range: -0.629 to 0.943) for Trial for each participant from the simplest GLMM on accuracy with only Trial as a fixed and random-effect.Although the total effect of State Anxiety and Uncontrollable Stress together was significant (p=.009), the mediation did not reach significance (p=.056).Thus, although both factors together impact learning, we cannot make strong inferences regarding a mediation effect.

Learning speed
For both tasks, we performed preregistered multiple linear regressions examining how the nine factors predicted learning speed (number of trials to learning defined by a preregistered criterion of 5 consecutive correct trials).We expected that Uncontrollable Stress would be associated with slower learning.However, none of the factors predicted slower learning in either task (Table S.7; S.8).

Supplementary tables Table S.1. Exploratory Factor Analysis (EFA) factor score test-retest Intraclass Correlations (ICCs) between session 1 and session 2 for scores from the seven-factor EFA (n = 49).
Confidence Intervals are 95%.The ICC estimates were remarkably low, and so we opted to use confirmatory factor models based on previously established scales and subscales instead.The power parameter was not recoverable, and so this model was not considered further.
, a four-factor model with State Anxiety (negatively-framed items), State Anxiety (positivelyframed items), Trait Anxiety (negatively-framed items), and Trait Anxiety (positively-framed items) was a better fit than a two-factor Trait and State Anxiety model.The State Anxiety negatively-framed and positively-framed factors significantly correlated with one another, r(425) = 0.645, p < .001;as did the Trait Anxiety negatively-framed and positively-framed factors, r(425) = 0.664, p < .001.Congeneric models were fitted using the "lavaan" package in R 20 .Model comparison was done using the "nonnest2" package in R 21 .20. Rosseel, Y. lavaan : An R Package for Structural Equation Modeling.J. Stat.Soft.48, (2012).21.Vuong, Q. H. Likelihood Ratio Tests for Model Selection and Non-Nested Hypotheses.

Figure S. 1 .
Figure S.1.Distribution of performance measures: learning slopes extracted from the null GLMM (z scored by default), the number of games learned, average trial to learning criterion for participants learning at least one game (N=398), and proportion of probabilistic errors for participants that experienced negative feedback at least once after reaching learning criterion (N=393).

Figure S. 2 .
Figure S.2.Distribution of sum scores for the state and trait anxiety as well as the perceived stress scale.The black lines indicate the mean of our sample for each scale.For the state and trait anxiety, the red line displays the cutoff that is used to indicate clinically significant symptoms of anxiety6 .We note that many of our participants scored above this threshold, which probably reflects increased levels of anxiety due to the covid pandemic that had just started when the data was collected.For the perceived stress scale, the yellow line indicate the separation between low and moderate stress, and the cyan line indicates the separation between moderate and high stress7 .

Figure S. 3 .
Figure S.3.Correlations between observed and generative parameter estimates for each of the three parameters of the winning hidden Markov model (pb r=0.6975; trans r=0.9096; qg r=0.8554)

Figure S. 5 .
Figure S.5.Confusion matrix showing the model frequencies obtained for synthetic data generated with different model for the reversal task.Models: ba = beta-alpha; baf = betaalpha-forget; bafc = beta-alpha-forget-conf; HMM3 = hidden Markov model with 3 free parameters; HMM4 = hidden Markov model with 4 free parameters.

Table S .
Table S.4.Results of the chi-squared ANOVA model comparisons comparing each of the nine single factor Generalised Logistic Mixed Models (GLMMs) for the reversal task to the null model (with only Trial as a fixed-effect, and by-subject random intercepts and slopes for Confidence intervals are 95%.se is the standard error of the log odds estimate.Significant fixed-effects and interactions are shown in bold.Log odds estimates can be transformed into odds ratios by exponentiating the value.p values are uncorrected for multiple comparisons.No effects Table S.7.Linear regression model results for the reversal task examining the effect of each factor on number of trials to learning (preregistered as five correct trials in a row), indexing learning speed.No factors significantly predicted learning speed.Confidence intervals are 95%.se is the standard error of the model estimate.Table S.8.Robustness check on the Generalised Logistic Mixed Model (GLMM) results for the reversal task using summed scores instead of congeneric factor scores: Results from the two separate models that are significant in table 1 with Summed Scores x Trial interactions.Confidence intervals are 95%.se is the standard error of the log odds estimate.* Indicates a significant effect.
2. Model comparisons of the fit of non-nested one and two-factor parallel andcongeneric models to obtain the factor scores (using the Vuong test in the "nonnest2" package in R).We used previously established scales (one-factor models) and subscales (two and fourfactor models; seeMcNeish & Wolf, 2020)to determine contributing items.Parallel models create linear transformations of the sum scores.Congeneric models produce weighted scores based on how related each item is to the construct.LRT is the likelihood ratio test statistic (values below 0 indicate the second model is a better fit; values above 0 indicate the first model is a better fit) and the p values test the significance of the LRT.BIC is Bayesian Information Criterion.CI lower and CI upper indicate 95% confidence intervals surrounding the difference between BICs for each model comparison.random intercepts and slopes for Trial.Confidence intervals are 95%.Log odds estimates can be transformed into odds ratios by exponentiating the value.The factor State Anxiety includes the negatively-framed items from the STAI-State questionnaire only.The factor Uncontrollable Stress includes the items from the subscale of the Perceived Stress Scale (also referred to in previous literature as Perceived Helplessness).Table S.6.Generalised Logistic Mixed Model (GLMM) results for the signalled task: The table shows results from the nine separate models with Factor Score x Trial interactions.

Table S .
9. Robustness check on the linear regression analysis on probabilistic error proportion using summed scores instead of congeneric factor scores: Results from the three models that are significant in table 3. Confidence intervals are 95%.se is the standard error of the model estimate.* indicates a significant effect.

Table S .
10. Linear regression model results for the signalled task examining the effect of each factor on number of trials to learning (preregistered as five correct trials in a row), indexing learning speed.No factors significantly predicted learning speed.Confidence intervals are 95%.se is the standard error of the model estimate.

Table S .
11. Model comparison for the signalled task using the Hierarchical Bayesian Inference toolbox (Piray et al., 2019).RLM refers to Reinforcement Learning Models; HMM refers to the hidden Markov model with three parameters.Model frequency indicates the ratio of

Table S .
12. Summary statistics (25 th , 50 th and 75 th percentiles) for the parameters from the winning hidden Markov model with three parameters, for the reversal task.The parameter p